Toward Qualitative Evaluation of Textual Entailment Systems
نویسندگان
چکیده
This paper presents a methodology for a quantitative and qualitative evaluation of Textual Entailment systems. We take advantage of the decomposition of Text Hypothesis pairs into monothematic pairs, i.e. pairs where only one linguistic phenomenon at a time is responsible for entailment judgment, and propose to run TE systems over such datasets. We show that several behaviours of a system can be explained in terms of the correlation between the accuracy on monothematic pairs and the accuracy on the corresponding original pairs.
منابع مشابه
Contradiction-focused qualitative evaluation of textual entailment
In this paper we investigate the relation between positive and negative pairs in Textual Entailment (TE), in order to highlight the role of contradiction in TE datasets. We base our analysis on the decomposition of Text-Hypothesis pairs into monothematic pairs, i.e. pairs where only one linguistic phenomenon at a time is responsible for entailment judgment and we argue that such a deeper inspec...
متن کاملCambridge: Parser Evaluation Using Textual Entailment by Grammatical Relation Comparison
This paper describes the Cambridge submission to the SemEval-2010 Parser Evaluation using Textual Entailment (PETE) task. We used a simple definition of entailment, parsing both T and H with the C&C parser and checking whether the core grammatical relations (subject and object) produced for H were a subset of those for T. This simple system achieved the top score for the task out of those syste...
متن کاملUsing Anaphora Resolution in a Question Answering System for Machine Reading Evaluation
This paper describes UAIC1’s Question Answering for Machine Reading Evaluation systems participating in the QA4MRE 2013 evaluation task. We submitted two types of runs, both type of runs based on our system from 2012 edition of QA4MRE, and both used anaphora resolution system. Differences come from the fact the textual entailment component was used or not. The results offered by organizer showe...
متن کاملEnhancing a Question Answering System with Textual Entailment for Machine Reading Evaluation
This paper describes UAIC’s Question Answering for Machine Reading Evaluation systems participating in the QA4MRE 2012 evaluation task. We submitted two types of runs, first type of runs based on our system from 2011 edition of QA4MRE, and second type of runs based on Textual Entailment system. For second types of runs, we construct the Text and the Hypothesis, asked by Textual Entailment syste...
متن کاملUIO-Lien: Entailment Recognition using Minimal Recursion Semantics
In this paper we present our participation in the Semeval 2014 task “Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment”. Our results demonstrate that using generic tools for semantic analysis is a viable option for a system that recognizes textual entailment. The invested effort in developing such tools allows us to ...
متن کامل